perf(PERF-27B-LMHEAD-FP4): keep the ModelOpt NVFP4 lm_head packed (#213) by localai-bot · Pull Request #263 · mudler/vllm.cpp

localai-bot · 2026-08-10T11:38:48Z

Row PERF-27B-LMHEAD-FP4 — issue #213 — spec
.agents/specs/perf-27b-lmhead-nvfp4.md.

The dense Qwen3.6 loader no longer dequantizes a ModelOpt NVFP4 lm_head to
BF16: it keeps the head packed and runs the logits GEMM on the same Marlin
W4A16 family vLLM pins for it. Commits 1-2 are the original spec and
implementation; commit 3 resolves round-1 review findings; commit 4 resolves
round-2, rebased onto origin/main a0fa12c7.

Scope, stated precisely: this changes the dense lm_head and nothing else.
Every other NVFP4 projection — MLP gate/up/down, attention o_proj, GDN
out_proj, MoE shared experts — is byte-unchanged on every backend.

Why this row matters: it is half the parity gap, and it is pure traffic

Operator decode-only nsys two-length diff (8 vs 136 tokens,
--cuda-graph-trace=node, both arms same tool, idle box) attributes the entire
27B deficit:

Component	ms/step
`lm_head` / Marlin	+8.6414
FP8 tower	+7.6068
splitK reduce	+0.0532
everything else	+1.0279
total	+17.3292 (vs measured +17.3292)

The nvidia checkpoint ships lm_head.weight as U8/NVFP4 (248320x2560 + FP8
scales = 0.6660 GiB) and vLLM runs it as its 129th Marlin launch. Dequantizing it
made us read 2.3683 GiB instead — +1.702 GiB/step of avoidable traffic,
11.183 ms/step, 11.3% of our step, as a lone
cutlass_80_tensorop_s16816gemm_bf16_128x64_32x6_nn_align2.

This refines the earlier framing: our Marlin is not slower than vLLM's.
Combined Marlin+lm_head is ours 54.511 ms for 11.3330 GiB (207.9 GiB/s) vs
vLLM 45.870 ms for 9.6309 GiB (210.0 GiB/s) — same kernel efficiency, we simply
moved more bytes. So this is a pure traffic win, the cleanest kind.

Measured (gate checkpoint `nvidia/Qwen3.6-27B-NVFP4`@`0893e160`, GB10)

Axis	Result
Peak host RSS	21.06 → 19.36 GiB (−1.70 GiB) — re-measurement OWED. The reading predates `ENG-LOAD-DIRECT-UPLOAD` (#150), which rewrote `LoadCtNvfp4Raw` to borrow mmap'd bytes and so changed what host RSS counts. Marked owed in BENCHMARKS/ENVIRONMENT/FEATURES/STATUS/NOW and the spec rather than carried as accepted.
Greedy continuations	byte-identical between our OWN packed and dequant arms — solid. This is an arm-vs-arm claim, NOT ours-vs-oracle: the oracle's own greedy is undetermined at ~8 of 32 positions on this corpus (top1-top2 margin exactly 0.000000; at one divergence its top five tokens were all exactly -2.651325), so cross-engine token diffs are judged against those margins, the ratified distributional case.
`test_qwen27_paged_engine`	235/235, unchanged
Throughput	packed 11.197/11.193 vs dequant 9.418/10.163 tok/s — indicative only: packed wins every leg and the packed legs agree to 0.04%, but the dequant legs disagree by 7.9%, so the direction is established and the magnitude is NOT. A binding grid is owed.

These CUDA numbers are the reviewer's and the operator's, quoted as theirs. No
CUDA gate was run in either fix round; a CUDA re-run on the changed head is
owed to the operator.

Round-1 findings resolved

MEDIUM-HIGH — a compressed-tensors-named NVFP4 head silently became W4A4
(alpha = 0.0078125, IsTrueW4A4() == true), contradicting the code's own
comment and docs/USAGE.md, taking the fp4-activation GEMM vLLM refuses for
an output head, and making the pre-capture Marlin build early-return. The head
is now W4A16 under both spellings unless VT_MODELOPT_W4A4=1, mirroring
ModelOptNvFp4W4A16LinearMethod, which deletes input_scale
(modelopt.py:1365, registered :1358 — re-verified against the pinned
oracle 555967922). The masking bug is fixed too: the model loader's head
probe accepts the CT naming, so a CT head is no longer read as
tie_word_embeddings.
MEDIUM — a backend without an fp4 GEMM dequantized the whole head on
EVERY forward call (~2.54 GB per step at the gate model, where the old code
paid it once at load). Nvfp4Weight gains a d_dequant_b resident built once
like d_packed, and the registry prepare hook builds it up front.
MEDIUM — the paged arms and the pre-capture build had no coverage. Both of
the reviewer's exact mutations now go RED.
MEDIUM — the unreachable if (lm_head_fp4.Empty()) guard in
PrepareBf16Resident is deleted; the RSS win is the loader no longer building
the f32 + bf16 arrays.
LOW-MED — "~2.3 GiB" corrected to the arithmetic and measured 1.70 GiB.
Records — Qwen3.6 NVFP4 baselines (27B and 35B-A3B) must reach vLLM speed parity #213 in the roadmap issue table, FEATURES.md cell made
self-consistent, BENCHMARKS.md entry with the indicative/solid split, spec
anchors and the IsQwen27QuantizedLinear deviation recorded, and the new test
moved out of test_qwen27_paged_engine's configuration block.

Round-2 findings resolved

BLOCKER — finding 2 above traded a per-call temporary for a lifetime
resident on every non-CUDA backend, for every NVFP4 projection. The caching
sat inside MatmulNvfp4{F32,Bf16}D, which also serve DenseMlpBlock
gate/up/down, attention o_proj, GDN out_proj and the MoE shared experts;
kMatmulNvfp4 is registered CUDA-only, so CPU, Vulkan, Metal, HIP and
Tenstorrent all take that fallback. Their steady state went from packed-only
to packed plus a bf16 expansion of the whole tower — roughly 4x the packed
bytes, on the backends where Vulkan on unified memory holds TWO copies of the weights: 27B peaks at 100.8 GiB RSS and OOM-reboots a Spark #203 already reports the 27B peaking at
100.8 GiB. CUDA is unaffected, which is why the CUDA gate could not see it.

Residency is now a property of the weight, not of the GEMM:
Nvfp4Weight::keep_dequant_b, default OFF, set by LoadDenseLmHead and by
nothing else. Per NVFP4 weight, steady state, on a backend with no fp4 GEMM:

packed bytes dequantized bf16 2*K*N

before this row resident per-call temporary

after round 1 resident resident, for every one

now resident resident for the head alone

Alternatives weighed and rejected (recorded in the spec): a per-backend
switch, and dequantizing the head into lm_head at prepare time (the prepare
hook may hold BORROWED const weights, so only mutable state is writable).

New coverage: the residency case now populates an NVFP4 tower as well as
an NVFP4 head and asserts no tower projection holds a d_dequant_b after two
forwards. RED under the mutation that removes the guard — 6 failing
assertions, 3 projections × 2 layers, exactly the round-1 behavior.
Overstated claim / seam divergence — the round-1 caching landed on
qwen3_5.cpp's private dispatcher while shared-seam
dense_nvfp4::MatmulNvfp4W4A16D still rebuilt K*N bf16 per call. That
parallel dispatcher predates this row: dense_nvfp4_gemm.h was extracted
from qwen3_5.cpp's anonymous namespace and its own SCOPE comment records
that the true-W4A4 path stays private there, so the two carry independent
Dev, MakeTensor, ResidentNvfp4 and DequantNvfp4ToBLayout copies.
Unifying them is a refactor this row does not do; the exception is now
recorded explicitly in the spec. The opt-in instead lives on the shared data
type Nvfp4Weight, which both dispatchers can read, defaulted OFF. They
can still disagree — dense_nvfp4_gemm.h:626-631 ignores the flag
outright and rebuilds the per-call temporary for every weight — they merely
do not, because no weight reachable from MatmulNvfp4W4A16D opts in today,
and the divergence direction is benign: the shared seam UNDER-caches, so it
cannot reintroduce the whole-tower expansion. Honoring the flag there belongs
to the unification refactor. This PR body and the commit messages claim only
the dense lm_head.
LOW — anchor drift in the file this row added: the input_scale delete
was cited as modelopt.py:1359-1362 at two sites; at the pinned oracle that
range is a blank line, the def and its docstring. Corrected to :1365.
LOW — docs/USAGE.md advertised K*N/2 + K*N/16 unqualified; it now
states that a backend with no fp4 GEMM pays the packed bytes plus one
bf16 operand, built at prepare, and that only the head is kept that way.
Judgement calls — MakePackedHead had external linkage and built its fp4
operands through MakeOwned's f32 branch (4 B/element over-allocation); it is
now internal and exactly sized. IsQwen27QuantizedLinear's "lm_head is never
quantized" rule is deliberately left alone: deleting it and its checked-in
expectation costs product review budget this PR does not have, and the
function still has zero production callers, so it moves no behavior.

Round-3 findings resolved

The load-bearing claim had zero coverage. keep_dequant_b is documented
as "set by LoadDenseLmHead and by nothing else", and the round-2 fix is
correct as written — but nothing enforced the scope. The reviewer added one
line to LoadNvfp4AnyNaming (qwen3_5_dense_weights.cpp:377):

Nvfp4Weight r;
r.keep_dequant_b = true;   // a second setter

and every gate stayed green: test_qwen27_dense_lmhead_fp4 6/6 1170/1170,
test_qwen27_paged_forward 20/20 702/702, test_mtp_speculator 13/13
171/171. That function is what every dense NVFP4 tower projection flows
through — MLP gate/up/down via LoadDenseMlp, attention q/k/v/o via
LoadAttnDense, GDN out_proj via LoadGdnDense — so the mutation reopens
the round-2 blocker verbatim. It escaped because the residency case built its
tower with MakeNvfp4Weight (direct struct construction, never a loader), and
because no test in the repo called LoadDenseMlp, LoadDenseAttn or
LoadQwen3_5DenseWeights at all. The CUDA gate is blind by construction:
kMatmulNvfp4 is registered on CUDA, so ResidentNvfp4DequantB is never
reached there.

This was the third instance of one pattern on this row — the code right at
every site it is tested at, its scope unpinned — so the new case pins the
invariant, not the call site. It loads a fully-NVFP4 dense layer of BOTH
layer types, under BOTH namings (ModelOpt weight_scale_2 and
compressed-tensors weight_packed), through the real LoadQwen3_5DenseLayer,
and sweeps every Nvfp4Weight the layer struct owns — fields, not call
sites — asserting the flag is false on all of them and true on the head. A
census assertion (11 routed projections per naming) stops a fixture that
silently stopped producing NVFP4 weights from reading as a pass.

RED mutation	failures
`keep_dequant_b = true` in `LoadNvfp4AnyNaming`'s ModelOpt arm (the reviewer's)	11
the same in `LoadCtNvfp4Raw` (the other naming arm)	11
a genuinely FOURTH setter, one line after `LoadDenseMlp`'s `down_proj` load	4

Each restored byte-for-byte; green after: 7/7, 1314/1314. The third
mutation is the one that matters for the ask: it is outside
LoadNvfp4AnyNaming, and the sweep still catches it.

LoadQwen3_5DenseLayer gains a has-taking overload declaration. The
resolver-only overload answers has with a constant true, which forces every
projection down the compressed-tensors spelling and so cannot reach the
ModelOpt arm at all. The definition already existed; only the declaration is
new, and it is the sole non-test edit in this round.

Overstated claim, corrected. The spec and this body said the shared
Nvfp4Weight opt-in means "the two dispatchers cannot disagree about a
weight". They can — dense_nvfp4_gemm.h:626-631 ignores the flag outright.
They merely don't, because nothing reachable there sets it, and the
disagreement direction is benign (the shared seam under-caches, never
over-caches). Reworded in the spec and above.

The CPU footprint direction was undisclosed. Every record carried only the
CUDA -1.70 GiB. On a no-fp4-GEMM backend the head holds 0.666 GiB packed
plus a 2.368 GiB bf16 operand. Arithmetic from the same K*N, now in
docs/USAGE.md, docs/BENCHMARKS.md and the spec:

Backend	before	after	delta
CUDA	2.368 bf16	0.666 packed	-1.70 GiB
Vulkan (unified, Vulkan on unified memory holds TWO copies of the weights: 27B peaks at 100.8 GiB RSS and OOM-reboots a Spark #203)	2.368 host bf16 + 2.368 device copy	0.666 packed + 2.368 device bf16	-1.70 GiB
plain CPU	2.368 bf16	0.666 + 2.368	+0.67 GiB

So Vulkan on unified memory holds TWO copies of the weights: 27B peaks at 100.8 GiB RSS and OOM-reboots a Spark #203's backend genuinely improves and plain CPU genuinely regresses by the
packed head's own bytes — paid once, instead of rebuilding 2.368 GiB on every
decode step as that backend did before this row.

A backend-asymmetric hard abort, now recorded. PrepareLmHeadResident
(qwen3_5.cpp:6534) aborts for any Qwen3_5DenseWeights whose lm_head_fp4
is non-empty but did not come from LoadDenseLmHead, and only on a backend
with no fp4 GEMM (CUDA returns at :6533 first). Unreachable in production, and
preferred to a silent fall-through because a head that quietly lost its opt-in
would rebuild 2.54 GB per step with no symptom but throughput.
tests/vllm/models/test_qwen27_paged_forward.cpp:1377 constructs exactly such
a weight and passes only because it never calls Prepare.

Merge with `origin/main`

Merged origin/main @848d4a87 (22 commits) rather than rebasing, so no pushed
commit is rewritten. Keyed records were resolved by taking the target-branch
version wholesale and reapplying this row's scoped edit — never an automatic
three-way combination:

.agents/NOW.md — main's shortened f32-out / invocation-parity / MiniMax rows
kept; the 27B key was byte-identical on base and main, so this row's edit
is reapplied verbatim (compressed by 3 characters to stay inside the
6,000-char budget, which main now sits exactly at).
docs/STATUS.md — main's 35B binding-grid rewrite (f500d1a6) kept; the 27B
key was likewise untouched by main, so this row's edit is reapplied verbatim.
include/vllm/model_executor/models/qwen3_5_weights.h — both sides ADD members
to Nvfp4Weight (keep_dequant_b/d_dequant_b here, ResidentSlot resident_marlin{,_pair} from CUDA BF16 MoE resident cache reuses stale pointers after engine teardown #237/fix(moe): the weights own their resident device state, not a static address map (#237) #276); union, neither edited.

A second merge then picked up a4f72f86 (waiver registry deleted) and
f323907e (KV-MOONCAKE-STORE spec). That one overlapped .agents/roadmap_v1.md's
issue table — by KEY, not by line: main added #287, this row adds #213.
Verified by key rather than accepted as an automatic combination.

After both merges the records differ from origin/main by exactly this row's
scoped edit: NOW.md 1 line, roadmap_v1.md 1, STATUS.md 1, BENCHMARKS.md
14, ENVIRONMENT.md 1, FEATURES.md 1, USAGE.md 28. The branch is level with
origin/main.

450a1b69 on main retired the per-class PR line budgets, so the earlier
"product 900/900" claim is moot and has been dropped.

Gates

Clean-configured CPU Release build (-Werror): 0 warnings, 0 errors.
Full ctest -j 1: 369/369, 0 failed. Round 2's single failure
(test_serve_low_tools) was pre-existing on main from The online-serving gate aborts on current main: harness looks for examples/server, the build produces examples/vllm-server #222 and is fixed by the
merge.
Focused: test_qwen27_dense_lmhead_fp4 7/7, 1314/1314 (was 6/6, 1170/1170),
test_qwen27_paged_forward 20/20, 702/702, test_mtp_speculator
13/13, 171/171.
scripts/agent-preflight.sh: all gates green (pre-edit, --staged, and
post-commit).
NO CUDA GATE WAS RUN in round 3, and none is owed by it — the round
touches no CUDA behavior. Round 3's own commit changes no file under src/;
its only non-test, non-record edit is a header declaration. The operator's
own run on cf9a3ba7 (test_qwen27_paged_engine 235/235 SACRED,
test_qwen27_paged_forward 702/702, test_qwen27_dense_lmhead_fp4 1170/1170,
test_qwen27_dense_forward 333/333, 0 skipped, FA2 marker present) stands.
Still owed to the operator, unchanged from round 2: the binding throughput grid
and the post-Model load / cold start time: measure it properly, then cut it #150 RSS re-measurement.

localai-bot · 2026-08-10T11:55:16Z

Operator verification — CUDA gate rerun independently

Per AGENTS.md ("The operator reruns the row's gate itself. An implementer or reviewer report is an input, never a gate result."), I rebuilt and regated this head myself rather than accepting the report.

Provenance. Head fa6ef3fe, transferred by git archive (never rsync), md5 8ed322ffa6baa4c44f52a48362ede4fc verified identical on both ends. Clean configure, sm_121a, Release, -DVLLM_CPP_CUTLASS_DIR=$HOME/cutlass-4.5.0 -DVLLM_CPP_TRITON=ON. Idle box, single flock $HOME/gpu.lock, ctest -j 1.

The build was verified non-degraded before gating — my script hard-aborts if the marker is missing, because a CUTLASS-less build silently omits FA2 and every result drifts:

-- CUTLASS found at /home/mudler/cutlass-4.5.0; enabling sm120a NVFP4 cutlass GEMM
-- FlashAttention-2 prefill/decode: ENABLED for arch(es) [121a]
build_exit=0

Result — 4/4 PASS:

1/4 Test  #32: test_qwen27_dense_forward ........   Passed    3.41 sec
2/4 Test  #75: test_qwen27_paged_forward ........   Passed   15.49 sec
3/4 Test #379: test_qwen27_paged_engine .........   Passed   48.49 sec   <- SACRED
4/4 Test #380: test_qwen27_dense_lmhead_fp4 .....   Passed    1.61 sec   <- the new gate
100% tests passed, 0 tests failed out of 4
focused_exit=0

So the SACRED 27B gate is intact and the new lmhead gate passes on a build I made from this exact head.

What this row is worth, measured

From the decode-only nsys two-length diff (8 vs 136 tokens, --cuda-graph-trace=node, both arms in the same tool, idle box, node-level tracing proven by integral launch counts) recorded on #213:

The checkpoint ships lm_head.weight as U8/NVFP4 (248320x2560 + FP8 scales = 0.6660 GiB). vLLM keeps it packed and runs it as its 129th Marlin launch. We dequantized it to BF16 and read 2.3683 GiB — +1.702 GiB/step of avoidable traffic, 11.183 ms/step, 11.3% of our step, as a lone cutlass_80_tensorop_s16816gemm_bf16_128x64_32x6_nn_align2.
Our Marlin is not slower than vLLM's. Combined Marlin+lm_head: ours 54.511 ms for 11.3330 GiB (207.9 GiB/s) vs vLLM 45.870 ms for 9.6309 GiB (210.0 GiB/s). Same kernel efficiency — we simply moved more bytes. This is a pure traffic win, and nobody should go looking for a Marlin tuning problem that does not exist.
It is 8.6414 ms of the 17.3292 ms/step deficit, which closes to four decimal places as lm_head 8.6414 + FP8 tower 7.6068 + splitK 0.0532 + other 1.0279.

The FP8 half is row/PERF-27B-GDN-FP8-QKVZ (spec pushed); together they are ~16 of the 17.33 ms.

Still owed before merge

A fresh scoped re-review of this head — the round-1 findings were fixed, and AGENTS.md requires the loop to run to PASS, not to stop at the fix. Dispatching that now.

Not claimed

I did not re-run the throughput A/B or the greedy continuation on this head. The throughput figures in the body remain INDICATIVE (packed faster in all four legs, packed legs agreeing to 0.04%, but the dequant legs disagreeing by 7.9%). Note also that cross-engine greedy text is not expected to be byte-identical here: the oracle's own greedy is undetermined at ~8 of 32 positions on this corpus (top1-top2 margin exactly 0.000000; at one divergence its top five tokens were all exactly -2.651325). Judge any token diff against the oracle's margins, not byte equality.

The dense loader's U8 branch dequantizes a ModelOpt NVFP4 lm_head into a BF16 [in,out] operand, so the logits GEMM re-reads ~2.543 GB every decode step where the packed head is ~0.715 GB. The transposed storage additionally forces a row-major NN GEMM with no nvjet_sm121 kernel, which is why an SM80 CUTLASS tile is selected on an sm_121a part. vLLM keeps the head quantized: ModelOptMixedPrecisionConfig.get_quant_method accepts ParallelLMHead and _quantized_layer_prefix_candidates appends the bare lm_head key, then ModelOptNvFp4W4A16LinearMethod pins MarlinNvFp4LinearKernel. Spec only; no implementation. Records that test_qwen27_paged_engine cannot see this path because its checkpoint ships a BF16 head. Refs #213 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode]

The dense loader's U8 branch dequantized a ModelOpt NVFP4 lm_head into a BF16 [in,out] operand, so the logits GEMM re-read ~2.543 GB every decode step where the packed head is ~0.715 GB, and the transposed [K,N] storage forced a row-major NN GEMM with no nvjet_sm121 kernel (an SM80 CUTLASS tile on an sm_121a part). vLLM keeps that head quantized: ModelOptMixedPrecisionConfig.get_quant_method accepts ParallelLMHead (modelopt.py:2508-2536) and _quantized_layer_prefix_candidates appends the bare `lm_head` key (modelopt.py:2491-2496), so ModelOptNvFp4W4A16LinearMethod — which pins MarlinNvFp4LinearKernel (modelopt.py:1249,1283-1284) and DELETES input_scale (modelopt.py:1359-1362) — resolves it, and logits_processor._apply_head (logits_processor.py:98-133) calls quant_method.apply every step. Verified against the pinned oracle at 555967922. LoadDenseLmHead routes an NVFP4 head through the SAME LoadNvfp4AnyNaming every other NVFP4 projection takes, into a new Qwen3_5DenseWeights::lm_head_fp4; the three dense consumers (eager ForwardDense, the gathered and non-gathered paged arms) and the dense MTP sibling now all select through one DenseLogitsF32D helper. The Marlin resident is built PRE-CAPTURE from the registry prepare hook (it Copies a host stack float; captured, that bakes a dangling stack address), and PrepareBf16Resident no longer stages a BF16 head owner. BF16, FP8, GGUF and tied heads are byte-unchanged, so every recorded unsloth benchmark is unaffected. VT_LMHEAD_FP4=0 is the same-binary rollback. test_qwen27_paged_engine cannot see this path — its checkpoint ships a BF16 head — hence the new synthetic loader+numerics gate. Refs #213 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode]

Six defects around the packed head, from the row's fresh review. The head itself is unchanged: it still loads packed, still runs the Marlin W4A16 logits GEMM, and the reviewer's hardware run stands (focused 5/5, test_qwen27_paged_engine 235/235, greedy continuations byte-identical packed vs dequant on nvidia/Qwen3.6-27B-NVFP4@0893e160, peak RSS 21.06 -> 19.36 GiB). 1. A compressed-tensors-named NVFP4 head silently became W4A4. LoadCtNvfp4Raw sets alpha from `input_global_scale` UNCONDITIONALLY — correct for a 27B TOWER projection, wrong for an output head — so a CT-named head came back with alpha=0.0078125 and IsTrueW4A4()==true, contradicting both the code's own comment and docs/USAGE.md. That head would take the fp4-activation GEMM vLLM refuses for it AND make the pre-capture Marlin build early-return. LoadDenseLmHead now drops the activation globals on BOTH spellings unless VT_MODELOPT_W4A4=1, mirroring ModelOptNvFp4W4A16LinearMethod, which deletes input_scale (modelopt.py:1365; the placeholder is registered at :1358 — verified against the pinned oracle 555967922). The trap was masked only by a second bug: the model loader probed `lm_head.weight`, so a pure-CT head was read as tie_word_embeddings. DenseCheckpointHasLmHead now accepts either naming, which is what USAGE.md already advertised. 2. Every backend without an fp4 GEMM dequantized the whole head on EVERY forward call. MatmulNvfp4{F32,Bf16}D's fallback built a fresh K*N bf16 operand per call; CPU registers only kMatmulNvfp4Fp4, and Vulkan/Metal register neither candidate. At the gate model that is ~2.54 GB of allocate-and-dequantize per step where the OLD code paid it once at load. Nvfp4Weight gains a `d_dequant_b` resident, built once exactly like `d_packed`, and the registry prepare hook builds it up front — so the same hook now covers both arms and is named PrepareLmHeadResident. 3. The paged arms had no coverage. The numerical case ran only the eager ForwardDense, so reverting either paged lm_head call site to the bf16 owner (an EMPTY OwnedTensor on a packed head) left the full CPU suite green, as did dropping the pre-capture build. Both mutations are now RED. 4. Unreachable dead code credited with the win. PrepareBf16Resident's `if (lm_head_fp4.Empty()) raw(DenseLmHead(weights))` could never take the false branch: its only caller is reached under IsPlainBf16Qwen3_5Dense, which this row made false whenever the head is packed, and `raw` already skips an empty tensor. Guard deleted. The RSS win comes from the LOADER no longer building the f32 + bf16 arrays, not from anything skipped at staging time. 5. "~2.3 GiB" was wrong. bf16 head 2,543,206,400 B = 2.368 GiB; packed K*N/2 + K*N/16 = 715,264,000 B = 0.666 GiB; delta 1.70 GiB, which is exactly the measured 21.06 -> 19.36 GiB. Corrected in the spec and ENVIRONMENT.md. The 0.715 GB / 2.543 GB byte figures were already right. 6. Records. #213 added to the roadmap issue table (the third of the three places that must agree). FEATURES.md no longer says "supported and gated" and "CUDA gate PENDING" in the same cell. BENCHMARKS.md records the A/B: RSS -1.70 GiB and byte-identical continuations SOLID; throughput 11.197/11.193 packed vs 9.418/10.163 dequant is INDICATIVE ONLY — packed wins every leg and the packed legs agree to 0.04%, but the dequant legs disagree by 7.9%, so the direction is established and the magnitude is NOT, and a binding grid is owed. The spec records the IsQwen27QuantizedLinear deviation (zero production callers, so the design's retirement would move no behavior), fixes the stale :506 anchor to :527, and fixes the modelopt.py:1359-1362 anchor to :1365. The new test also no longer splits test_qwen27_paged_engine's configuration block. Also corrected: the capture-safety comment overstated the failure mode. Building the Marlin resident inside a capture would abort the capture with an error, not silently bake a dangling address; the build still belongs before capture. A BENCHMARKS measurement owes STATUS.md and NOW.md under the doc-checkpoint rule, and both pages were at zero headroom. Neither budget is widened and no checker is touched: the 27B-NVFP4 STATUS cell pays for its new fact by dropping two asides docs/BENCHMARKS.md states authoritatively — the "(ModelOpt FP8 tower)" parenthetical and the "decode ~100% GPU-busy" diagnostic, both on its Roof row — leaving that page byte-for-byte the same length, so its shrink-only ratchet stays byte-tight and untouched; the NOW row is compacted in place. Every measured number and binding claim is kept verbatim. Refs #213 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode]

Round-2 review findings on the packed NVFP4 head. The head itself is unchanged: it still loads packed and still runs the Marlin W4A16 logits GEMM on CUDA. 1. BLOCKER. The round-1 fix traded a per-call temporary for a LIFETIME resident on every non-CUDA backend, for every NVFP4 projection -- not just the head. The caching sat inside MatmulNvfp4{F32,Bf16}D, which also serve DenseMlpBlock gate/up/down, attention o_proj, GDN out_proj and the MoE shared experts. kMatmulNvfp4 is registered CUDA-only (cuda_matmul_nvfp4.cu), so CPU, Vulkan, Metal, HIP and Tenstorrent ALL take that fallback: their steady state went from packed-only to packed plus a bf16 expansion of the whole tower, roughly 4x the packed bytes, on the backends where issue #203 already reports the 27B peaking at 100.8 GiB and OOM-rebooting a Spark. CUDA is unaffected, which is why the CUDA gate could not see it. Residency is now a property of the WEIGHT, not of the GEMM: Nvfp4Weight::keep_dequant_b, default OFF, set by LoadDenseLmHead and by nothing else. It is worth its bytes exactly where one operand is re-read whole every step and there is one of it -- the output head. MatmulNvfp4DequantB carries the branch, so a weight that did not opt in keeps the PER-CALL temporary it had before this row, byte for byte. Alternatives weighed and rejected, recorded in the spec: a per-BACKEND switch (the tower is the problem on every fallback backend, not on one), and dequantizing the head into Qwen3_5DenseWeights::lm_head at prepare time (the prepare hook may hold BORROWED const weights, so only mutable residency state is writable there). Footprint on a no-fp4-GEMM backend, per NVFP4 weight, steady state: before this row packed; per-call K*N*2 temporary round 1 packed + K*N*2 FOR EVERY ONE now packed, and K*N*2 for the HEAD ALONE Coverage that catches it: the prepare/residency case now populates an NVFP4 TOWER as well as an NVFP4 head and asserts that after two forwards no tower projection holds a d_dequant_b. RED under the mutation that removes the keep_dequant_b guard -- 6 failing assertions, 3 projections x 2 layers, which is exactly the round-1 behavior. 2. The claim was overstated, and it deepened a seam divergence. The caching landed on qwen3_5.cpp's PRIVATE dispatcher while the shared-seam dense_nvfp4::MatmulNvfp4W4A16D still rebuilt K*N bf16 per call. That parallel dispatcher predates this row -- dense_nvfp4_gemm.h was EXTRACTED from qwen3_5.cpp's anonymous namespace and its SCOPE comment records that the true-W4A4 path stays private there, so the two also carry independent Dev, MakeTensor, ResidentNvfp4 and DequantNvfp4ToBLayout copies. Unifying them is a refactor this row does not do; the exception is now recorded explicitly in the spec. What this row does instead is put the opt-in on the SHARED data type (Nvfp4Weight) that both dispatchers read, defaulted OFF, so they cannot disagree about a weight and no weight reachable from MatmulNvfp4W4A16D opts in today. This change covers the dense lm_head, and says so. 3. Anchor drift, in the file this row added. The input_scale delete was cited as modelopt.py:1359-1362 at two sites in the test; at the pinned oracle 555967922 that range is a blank line, the def and its docstring. Corrected to :1365 (the register stays :1358), re-verified against the oracle checkout. 4. docs/USAGE.md advertised "K*N/2 + K*N/16 instead of 2*K*N" unqualified. It now states that a backend with no fp4 GEMM pays the packed bytes PLUS one bf16 operand, built at prepare, and that only the head is kept that way. 5. Rebased onto origin/main a0fa12c (40 commits, incl. ENG-LOAD-DIRECT-UPLOAD #150). Keyed records verified against the target-branch version: NOW.md, roadmap_v1.md, STATUS.md and BENCHMARKS.md differ from origin/main by exactly this row's scoped edit and nothing else. Because #150 rewrote LoadCtNvfp4Raw to BORROW mmap'd bytes, the 21.06 -> 19.36 GiB reading was taken under different RSS accounting; it is now marked OWED a re-measurement in BENCHMARKS, ENVIRONMENT, FEATURES, STATUS, NOW and the spec rather than carried as accepted. No hardware was available to this session to re-take it. Also fixed, raised as judgement calls: the test helper MakePackedHead had external linkage (it sat outside its file's anonymous namespace) and built its fp4 operands through MakeOwned's f32 branch, over-allocating 4 B/element; it is now internal and exactly sized. IsQwen27QuantizedLinear's "lm_head is never quantized" rule is deliberately left alone -- deleting it and its checked-in expectation costs review budget this PR does not have, and the function still has zero production callers, so it moves no behavior. It is recorded in the spec. CPU Release ctest -j 1: 366/367. The one failure, test_serve_low_tools (test_online_gate_client), is PRE-EXISTING on main from the just-landed BENCH-SERVER-BINARY-RENAME work: tools/, tests/tools/, benchmarks/ and scripts/ are byte-identical to origin/main in this tree, so nothing here can reach it. Focused: test_qwen27_dense_lmhead_fp4 6/6, test_qwen27_paged_forward 20/20, test_mtp_speculator 13/13. check-pr-size: product 900/900. NO CUDA GATE WAS RUN in this session -- dgx.casa was not reachable -- so the CUDA rerun and the RSS re-measurement are owed to the operator. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode]

Brings the row onto `origin/main` @848d4a87 (22 commits, including #237/#276's `ResidentSlot` members on `Nvfp4Weight`, #150's follow-ups, the docs site, and 450a1b6 retiring the per-class PR line budgets). Keyed records were resolved by taking the TARGET BRANCH version wholesale and reapplying this row's scoped edit, never a three-way combination: .agents/NOW.md main's 27B line kept ("0.72x -> 0.85x", and main's shortened f32-out/invocation-parity/MiniMax rows), with this row's "#213 head packed" + next-step reapplied. docs/STATUS.md main's 27B and 35B lines kept verbatim (the 35B binding grid rewrite from f500d1a is main's, not ours), with the packed-lm_head clause reapplied to the 27B row alone. qwen3_5_weights.h both sides ADD to `Nvfp4Weight`; union, no edit to either. After the merge the four keyed records differ from `origin/main` by exactly this row's scoped edit: NOW.md 1 line, roadmap_v1.md 1 line, STATUS.md 1 line, BENCHMARKS.md 13, plus ENVIRONMENT.md 1, FEATURES.md 1, USAGE.md 23. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode]

Round-3 review finding, test-only on the product side. The head is unchanged: it still loads packed and still runs the Marlin W4A16 logits GEMM on CUDA. The only non-test edit is a header DECLARATION of a function that already existed. 1. FINDING. `Nvfp4Weight::keep_dequant_b` is documented as "set by LoadDenseLmHead and by nothing else", and the round-2 fix is correct as written -- but nothing enforced the scope. The reviewer added ONE line to `LoadNvfp4AnyNaming` (qwen3_5_dense_weights.cpp:377): Nvfp4Weight r; r.keep_dequant_b = true; // a second setter and every gate stayed green: test_qwen27_dense_lmhead_fp4 6/6 1170/1170, test_qwen27_paged_forward 20/20 702/702, test_mtp_speculator 13/13 171/171. That function is what every dense NVFP4 TOWER projection flows through -- MLP gate/up/down via LoadDenseMlp, attention q/k/v/o via LoadAttnDense, GDN out_proj via LoadGdnDense -- so the mutation reopens the round-2 blocker verbatim: a lifetime bf16 expansion of the whole tower on CPU, Vulkan, Metal, HIP and Tenstorrent. It escaped for two reasons: the residency case built its tower with MakeNvfp4Weight (direct struct construction, never a loader), and NO test in the repo called LoadDenseMlp, LoadDenseAttn or LoadQwen3_5DenseWeights at all. The CUDA gate is blind by construction -- kMatmulNvfp4 is registered on CUDA, so ResidentNvfp4DequantB is never reached there. This was the THIRD instance of one pattern on this row: the code correct at every site it is tested at, its SCOPE unpinned. So the new case pins the INVARIANT, not the call site. It loads a fully-NVFP4 dense layer of BOTH layer types, under BOTH namings, through the real `LoadQwen3_5DenseLayer`, and sweeps every Nvfp4Weight the layer STRUCT owns -- fields, not call sites -- asserting the flag is false on all of them and true on the head. A census assertion (11 routed projections per naming) keeps a fixture that stopped producing NVFP4 weights from reading as a pass. RED, three mutations, each restored byte-for-byte afterwards: M1 keep_dequant_b in LoadNvfp4AnyNaming's ModelOpt arm 11 failed M2 keep_dequant_b in LoadCtNvfp4Raw 11 failed M3 a FOURTH setter, one line after LoadDenseMlp's down_proj load -- outside LoadNvfp4AnyNaming 4 failed GREEN restored: 7/7, 1314/1314. `LoadQwen3_5DenseLayer` gains a `has`-taking overload in the header. The resolver-only overload answers `has` with a constant true, which forces every projection down the compressed-tensors spelling and cannot reach the ModelOpt arm at all. The definition already existed; only the declaration is new. 2. OVERSTATED CLAIM. The spec said the shared `Nvfp4Weight` opt-in means "the two dispatchers cannot disagree about a weight". They can: dense_nvfp4_gemm.h:626- 631 ignores the flag outright and rebuilds the per-call temporary for every weight. They merely do not, because nothing reachable there sets it -- and the divergence direction is benign, the shared seam UNDER-caches and so cannot reintroduce the whole-tower expansion. Reworded to that. 3. THE CPU FOOTPRINT DIRECTION WAS UNDISCLOSED. Every record carried only the CUDA -1.70 GiB. On a no-fp4-GEMM backend the head holds 0.666 GiB packed PLUS a 2.368 GiB bf16 operand. Arithmetic from the same K*N, now stated in docs/USAGE.md, docs/BENCHMARKS.md and the spec: CUDA 2.368 -> 0.666 -1.70 GiB Vulkan 2.368 + 2.368 -> 0.666 + 2.368 -1.70 GiB CPU 2.368 -> 0.666 + 2.368 +0.67 GiB So #203's backend genuinely improves and plain CPU genuinely regresses by the packed head's own bytes, paid once instead of rebuilding 2.368 GiB per step. 4. A BACKEND-ASYMMETRIC HARD ABORT, now recorded in the spec. PrepareLmHeadResident (qwen3_5.cpp:6534) aborts for any Qwen3_5DenseWeights whose lm_head_fp4 is non-empty but did not come from LoadDenseLmHead, and only on a backend with no fp4 GEMM (CUDA returns at :6533 first). Unreachable in production; tests/vllm/models/test_qwen27_paged_forward.cpp:1377 constructs exactly such a weight and passes only because it never calls Prepare. The -1.70 GiB CUDA figure stays marked OWED, not carried: #150 rewrote LoadCtNvfp4Raw to borrow mmap'd bytes and changed the RSS accounting. Gates: clean CPU Release build, 0 warnings 0 errors. Focused test_qwen27_dense_lmhead_fp4 7/7 1314/1314 (was 6/6 1170/1170), test_qwen27_paged_forward 20/20 702/702, test_mtp_speculator 13/13 171/171. Full `ctest -j 1`: 369/369, 0 failed (round 2's one failure, test_serve_low_tools, was pre-existing on main and is fixed by the merge). scripts/agent-preflight.sh: all gates green. NO CUDA GATE WAS RUN and none is owed by this change -- it touches no CUDA behavior; the operator's run on cf9a3ba (paged_engine 235/235, paged_forward 702/702, dense_forward 333/333) stands. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode]

Two commits landed on main during round 3: `a4f72f86` (the waiver registry is deleted; a commit documents its own exception) and `f323907e` (the KV-MOONCAKE-STORE verdict spec). Neither touches this row's code. One keyed record overlapped, `.agents/roadmap_v1.md`'s issue table, and the overlap is by KEY, not by line: main added the `#287` row, this row adds `#213`. Verified by key rather than accepted as an automatic combination — after the merge the table differs from `origin/main` by exactly the one `#213` line, with main's `#287` row intact. `scripts/agent-preflight.sh`: all gates green on the merged tree, including the checker set as it stands after the waiver-registry deletion. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode]

… broke (#298) Merges the DSR repair. main has been RED on device-leakage since the PERF-27B-LMHEAD-FP4 landing: PrepareLmHeadResident opened a NEW `#ifdef VT_MARLIN_NVFP4` in the device-agnostic shared layer, taking the vt_ifdef bucket to 33 against a baseline of 32. Verified both directions myself before merging, rather than on report: origin/main -> `DSR by bucket: ... vt_ifdef=33 -> total 33`, exit 1; this branch -> `OK (DSR 32 == baseline 32, ratchet holds)`, exit 0. WHY IT LANDED RED, which is the part worth keeping. The device-leakage job never once finished on PR #263 -- every run was superseded in the queue before leaving `pending` -- and the checker is NOT in agent-preflight.sh, so no local gate covered it either. A regression therefore reached main through a gap between two verification surfaces rather than past either of them. Adding this checker to preflight would close it; that is outside this row and is left named rather than silently fixed. The repair MOVES the gate rather than widening it: the checker forbids both growing the DSR-ALLOW list and raising the baseline, so BuildDenseHeadMarlinResident now carries the guard inside the region that already owns that kernel family, with an `#else` stub. Same three terms, same short-circuit order, negated as a whole; behaviour unchanged. PR #300 CI: 13 checks pass, including cuda-fat-build at 1h34m, which is what proves the `#ifdef` arm actually compiles -- it could not be verified locally, since VT_MARLIN_NVFP4 needs CUDA arch 12.0a/12.1a. The single failing check, sanitize-cpu (address,undefined), fails IDENTICALLY on main with the same five tests in files this row does not touch. It is filed as #301: a UBSan unaligned uint16_t (bf16) load class, most plausibly from #150 making loaders borrow mmap'd bytes rather than copying into aligned buffers. It is not this row's, and that lane has been red long enough that a green sanitize-cpu is not currently a signal any PR can produce. Closes #298 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode]

… ratchet (mudler#298) `origin/main` is RED on the `device-leakage` CI job, and has been since `bc2fa7d9` landed PERF-27B-LMHEAD-FP4 (mudler#213, PR mudler#263). It went unnoticed because the job never finished on any push of that PR -- every run was superseded in the queue before it left `pending` -- and because the ratchet needs neither CUDA nor a GPU yet is not part of `scripts/agent-preflight.sh`, so no local gate the row ran covered it either. ERROR: DSR REGRESSION in bucket 'vt_ifdef': 33 > baseline 32. `PrepareLmHeadResident` opened a NEW `#ifdef VT_MARLIN_NVFP4` in the device-agnostic shared layer to guard the Marlin build. That is exactly the drift `scripts/check-device-leakage.py` exists to stop: the shared layer's build-time kernel-feature gates may only decrease, per bucket, and `src/vllm/model_executor/models/qwen3_5.cpp` went 26 -> 27. Verified as introduced by that row, not pre-existing: `origin/main` @`f323907e`, the commit immediately before the landing, reports `vt_ifdef=32 -> total 32, ratchet holds`. Current `origin/main` reports 33 and exits 1. The checker's own message says to repair the code and never to grow the DSR-ALLOW list or raise the baseline, so the gate moves rather than gets excused. `BuildDenseHeadMarlinResident` now carries it, inside the `#ifdef VT_MARLIN_NVFP4` region that already owns this kernel family, with an `#else` stub returning false. The call site becomes one ordinary `if`. BEHAVIOR IS UNCHANGED. The predicate is the same three terms in the same short-circuit order, negated as a whole: before #ifdef VT_MARLIN_NVFP4 if (!IsTrueW4A4() && MarlinMoeEnabled() && OpRegistered(...)) { BuildMarlinDenseResident(...); Synchronize(); return; } #endif after if (IsTrueW4A4() || !MarlinMoeEnabled() || !OpRegistered(...)) return false; // -> hook falls through BuildMarlinDenseResident(...); Synchronize(); return true; `OpRegistered` is asked with `d.q.device.type`, which is `queue.device.type` -- `Dev` is constructed from that same queue one line above the old guard. Also here, and owed by PR mudler#263's round-3 finding 2 rather than by this one: `docs/FEATURES.md` still carried the `-1.70 GiB` unqualified. It is CUDA's number, and that page is one of the records the finding named. Now marked, and the cell re-fitted under the 220-char table limit. Gates: `check-device-leakage` 32 == baseline 32, ratchet holds, and its own mutation suite 26/26. CLEAN-CONFIGURED CPU Release rebuild (fresh build dir, the incremental-masks-Werror trap): 0 warnings, 0 errors. Full `ctest -j 1` 369/369, 0 failed. Focused: test_qwen27_dense_lmhead_fp4 7/7 1314/1314, test_qwen27_paged_forward 20/20 702/702, test_mtp_speculator 13/13 171/171, test_qwen27_dense_forward 7/7 333/333. `scripts/agent-preflight.sh` green. NOT VERIFIED HERE: the `#ifdef` arm. VT_MARLIN_NVFP4 requires CUDA arch 12.0a/12.1a (CMakeLists.txt:473,1618), so a CPU box compiles only the `#else` stub. CI's `cuda-fat-build` is what proves the Marlin arm still compiles, and the operator's CUDA gate is what proves it still runs. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode]

localai-bot mentioned this pull request Aug 10, 2026

Qwen3.6 NVFP4 baselines (27B and 35B-A3B) must reach vLLM speed parity #213

Open

mudler added 4 commits August 10, 2026 12:40

localai-bot force-pushed the row/PERF-27B-LMHEAD-FP4 branch from fa6ef3f to cf9a3ba Compare August 10, 2026 13:33

mudler added 3 commits August 10, 2026 15:56

localai-bot merged commit bc2fa7d into main Aug 10, 2026
13 of 15 checks passed

This was referenced Aug 10, 2026

main is RED on the device-leakage DSR ratchet: PrepareLmHeadResident added a 33rd vt_ifdef #298

Closed

fix(PERF-27B-LMHEAD-DSR): main is red on the DSR ratchet — move the gate, do not excuse it (#298) #300

Merged

Navigation Menu

perf(PERF-27B-LMHEAD-FP4): keep the ModelOpt NVFP4 lm_head packed (#213) - #263

perf(PERF-27B-LMHEAD-FP4): keep the ModelOpt NVFP4 lm_head packed (#213)#263
localai-bot merged 7 commits into
mainfrom
row/PERF-27B-LMHEAD-FP4

localai-bot commented Aug 10, 2026 •

edited

Loading

Uh oh!

localai-bot commented Aug 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	packed bytes	dequantized bf16 `2KN`
before this row	resident	per-call temporary
after round 1	resident	resident, for every one
now	resident	resident for the head alone

Conversation

localai-bot commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why this row matters: it is half the parity gap, and it is pure traffic

Measured (gate checkpoint nvidia/Qwen3.6-27B-NVFP4@0893e160, GB10)

Round-1 findings resolved

Round-2 findings resolved

Round-3 findings resolved

Merge with origin/main

Gates

Uh oh!

localai-bot commented Aug 10, 2026

Operator verification — CUDA gate rerun independently

What this row is worth, measured

Still owed before merge

Not claimed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

localai-bot commented Aug 10, 2026 •

edited

Loading

Measured (gate checkpoint `nvidia/Qwen3.6-27B-NVFP4`@`0893e160`, GB10)

Merge with `origin/main`